System and method for automated electronic catalogue management and electronic image quality assessment

ABSTRACT

In various examples, a system receives image data characterizing an image of an item. Additionally, the system implements a first set of operations and a second set of operations. In some examples, the first set of operations includes performing a structural similarity analysis of the item, based on the image data, and determining a structural similarity score based on the structural similarity analysis of the item. In other examples, the second set of operations includes generating a plurality of derivative images by applying a plurality of distortions to the image of the item, extracting one or more features based at least on the plurality of derivative images, and determining the quality of the image based at least on the extracted one or more features and the structural similarity score.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Pat. Application No.17/493,417, filed Oct. 4, 2021, now U.S. Pat. No. [______], which is acontinuation U.S. Pat. Application No. 16/548,162, filed Aug. 22, 2019,now U.S. Pat. No. 11,164,300, which claims benefit of priority to U.S.Provisional Pat. Application No. 62/778,962 filed Dec. 13, 2018 andIndian Provisional Application No. 201811031632, filed Aug. 23, 2018,each of which is hereby incorporated by reference in their respectiveentireties.

TECHNICAL FIELD

The present disclosure relates to automated electronic cataloguemanagement and electronic image quality assessment, and morespecifically to using automated electronic image quality assessment tocatalogue and re-order images.

BACKGROUND

Catalogue management is a very important aspect in e-commerce as ithelps the visitors to websites efficiently select respective items. Inevery retail website, the items displayed are in a particular orderbased on their respective categories. For items which have more than oneview, the order of the views is predetermined based on the itemclassification. However, when receiving images/photographs of the item,there are often several problems with the received information. First,there is a question of identifying the item: do the images (orassociated metadata) predefine the object in question? If not, how isthe object identified?

Second, there is a question of quality: do the images meet the requiredquality for display on an official website? If being performed by ahuman being, such determinations may lack accuracy due to the subjectiveability of human visual analysis. Third, using current technology, theimages must be manually ordered according to the object’sclassification, which again relies on the inaccuracy of human incompleting the task.

TECHNICAL PROBLEM

How to train a computer to correctly identify and classify an objectwhen electronic images of the object have distinct orientations andquality.

SUMMARY

Additional features and advantages of the disclosure will be set forthin the description which follows, and in part will be obvious from thedescription, or can be learned by practice of the herein disclosedprinciples. The features and advantages of the disclosure can berealized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims. These and otherfeatures of the disclosure will become more fully apparent from thefollowing description and appended claims, or can be learned by thepractice of the principles set forth herein.

An exemplary method performed according to the concepts disclosed hereincan include: receiving a plurality of images of an item; identifying,via a processor configured to perform image analysis, and within eachimage in the plurality of images, the item; performing, via theprocessor, a structural similarity analysis of the item, to yield astructural similarity score; for each image in the plurality of imagesapplying, via the processor, a plurality of distortions, such that foreach image in the plurality of images a plurality of distorted imagesare generated; identifying, via the processor, within the plurality ofdistorted images associated with each image in the plurality of images,at least one feature; and applying, via the processor, a regressionmodel to the plurality of images using the at least one feature and thestructural similarity score.

An exemplary system configured according to the concepts disclosedherein can include: a processor configured to perform image analysis;and a computer-readable storage medium having instructions stored which,when executed by the processor, cause the processor to performoperations including: receiving a plurality of images of an item;identifying, within each image in the plurality of images, the item;performing a structural similarity analysis of the item, to yield astructural similarity score; for each image in the plurality of imagesapplying a plurality of distortions, such that for each image in theplurality of images a plurality of distorted images are generated;identifying within the plurality of distorted images associated witheach image in the plurality of images, at least one feature; andapplying a regression model to the plurality of images using the atleast one feature and the structural similarity score.

An exemplary non-transitory computer-readable storage medium configuredaccording to this disclosure can have instructions stored which, whenexecuted by a computing device configured to perform image processing,cause the computing device to perform operations including: receiving aplurality of images of an item; identifying, within each image in theplurality of images, the item; performing a structural similarityanalysis of the item, to yield a structural similarity score; for eachimage in the plurality of images applying a plurality of distortions,such that for each image in the plurality of images a plurality ofdistorted images are generated; identifying within the plurality ofdistorted images associated with each image in the plurality of images,at least one feature; and applying a regression model to the pluralityof images using the at least one feature and the structural similarityscore.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a first example method embodiment;

FIG. 2 illustrates an exemplary flowchart of a disclosed process;

FIG. 3 illustrates an exemplary convolutional neural networkarchitecture for image orientation classification;

FIG. 4 illustrates a second exemplary method embodiment; and

FIG. 5 illustrates an example method embodiment.

DETAILED DESCRIPTION

Various embodiments of the disclosure are described in detail below.While specific implementations are described, it should be understoodthat this is done for illustration purposes only. Other components andconfigurations may be used without parting from the spirit and scope ofthe disclosure. This disclosure is directed to automated electroniccatalogue management, electronic image quality assessment, andselecting/ordering electronic images for use in an electronic cataloguebased upon the assessments made. More specifically, the solutionsdisclosed provide for an algorithm (using applications of computervision and deep learning) which can automatically identify the variouscomplex orientations of the catalogue image and sort it accordingly.Next step an algorithm which can detect the quality of the catalogueimages (using a structural similarity metric and/or deep learning), suchthat quality can be predicted for images without a reference image orsubject.

Current electronic image classification practices often rely onrule-based methods to classify image orientations, which is prone toerrors and not robust. Likewise, when a histogram of gradients iscurrently used in image classification, the accuracy achieved isinsufficient for quality classification. In addition, with regards tothe image quality assessment, metrics such as Signal to Noise Ratio(SNR), mean squared error, etc. are often used. Again, such metrics failto provide adequate results and human perceived quality.

Tasks associated with four phases of the operations are disclosed anddescribed. These tasks/phases can be combined, excluded, or otherwiseused as required for any specific embodiment.

Phase 1 - Image Orientation Classification

To classify an image into one of a predetermined set of classifications(such as Front, Side, and Back views), the image can be compared tostatistical markers of previously categorized images. That is, multipleimages can be analyzed, their features extracted, and a histogram ofgradient features used in those images can be generated showingpredictions of features for objects in a known orientation. For example,multiple shirts having a front view can be analyzed, and the system canidentify left and right sleeves within the images, as well as a “V” or“swoop” where a shirt neckline appears. By contrast, the angle of animage a side view may result in a significantly smaller proportion of animage being associated with the neckline. By comparing the features of anew image to features of known images, a histogram of gradient featuresof the new image can be identified as predictors. A histogram ofgradient based features are robust features which, for example, givedirection of the color gradient within an image, such that the histogramof gradient based features differ between different images. Thesepredictors can provide statistical estimates of how similar the newimage, or portions of the image, are to known images (or portions of theknown images).

The predictors can also be used as inputs into a Convolution NeuralNetwork (CNN) model trained to identify the distinct classifications ofimages. In one configuration, pre- trained embeddings from a predefinedmodel are extracted, and using the extracted features a machine learningmodel is formed which in turn generates the CNN model. The CNN modeluses a cross-correlation of the predictors with known features fromprevious images to identify common aspects between the known featuresand the current image being analyzed. In one example, the CNN model (orportions of the model extracted) can be split into a 3×3 depth-wiseconvolution and a 1×1 point-wise convolution, which increases accuracyand speed. In some cases, a logistic regression model (trained onsimilar data as the CNN model) can be combined with the CNN model(and/or other models) as part of the image orientation classification,which can further increase the overall object (and its orientation)recognition.

Phase 2 - Image Quality Assessment

To assess the quality of an image, traditional (subjective) assessmentsby human beings does not result in a repeatable accuracy. To counterthis, systems configured according to this disclosure can use structuralsimilarity to identify the object, then determine the quality of theobject identified. Metrics to determine the similarity can include peaksignal-to-noise ratio (PSNR) and the mean squared error (MSE), which canoperate directly from the intensity of the image. However, such metricsfail to account for how a human being would perceive the image. Toaccount for human perception, the system utilizes a structuralsimilarity index to which the object is compared, where the structuralsimilar index can take into account the impact of changes inluminescence, contrast, and structure within the image being considered.The structural similarity index can be a single score which takes intoaccount all of the individual factors (such as luminescence, contrast,etc.).

To further assess the image, a methodology 1 s needed which can operatewithout reference images, or alternatively with a very small data set.Instead, the architecture and data pipeline described makes the modelindependent of a reference image, such that when an image is received,the system extracts the quality embeddings from the architecture, whichserves as an input to the ridge regression model to predict thestructural similarity score (which in tum indicates the quality of theimage). To accomplish this, distortion is added to the one or moreimages, resulting in the original (non-distorted) images and distortedimages of the object. If a particular configuration is using referenceimages, the reference images can be distorted using the samedistortion/noise algorithms (such as mean blur, Gaussian blur, bilateralblur, median blur, etc.). Table I illustrates an example of how a sampleimage can be distorted and the resulting classes. The features, in thisexample, are extracted initially from pre-defined embeddings which havebeen trained with dense layers in classifying the quality of images,thereby forming quality embeddings.

TABLE 1 Type of Noise added Kernels and Parameters No Of ClassesReference Image - 1 Mean Blur (5,5),(25,25),(55,55),(75,75) 4 GaussianBlur (5,5),(25,25),(55,55),(95,95) 4 Bilateral Blur(9,50,50),(9,125,125) 2 Median Blur 5,27 2

A comparison between the original images and the distorted images (aswell as distorted reference images, if available) can then result infeatures related to quality characteristics of the images beinganalyzed.

Once the quality related features have been identified/extracted fromthe images, a structural similarity score for the distorted images iscomputed. This structural similarity score can, when using referenceimages, also be based on the reference images (including distortedreference images). The quality related features can be taken aspredictor variables, and the structural similarity score for one or moreof the images can be used as response variables. A ridge regression (atechnique for analyzing multiple regression data that has collinearity)model can be used with the structural similarity score. Ridge regressionmodel is regularized regression which, in this example, containsadditional constraints on the model parameters which are not present inlinear regression. L2 regularization is used in ridge regressions. Theregularization parameter has been determined intelligently suitable forthe problem.

In other words, when a new image is detected, features can be extractedfrom the image. These features can be used as a test data point for theridge regression model while a structural similarity score is generated(in parallel or in series) for that image. The system can determine,based on a business unit to which the image(s) will be assigned, cutoffsfor the similarity score and/or test data to determine if the imagequality is at an acceptable, predetermined quality, or if the imagesmust be revised or otherwise corrected.

These variations shall be described herein as the various embodimentsare set forth. The disclosure now turns to FIG. 1 . FIG. 1 illustrates afirst example method embodiment. In this example, a supplier sharesimages of items (102) with the purpose of the receiving entitypublishing the images in such a way that consumers can view the imageson the Internet. For example, the supplier can supply an onlinemarketplace with a generic, “stock” photograph of the item to be sold.The system uses deep learning and computer vision to perform qualitybased filtering (104) on the images received. More specifically, thesystem uses a processor, specifically configured to perform imageprocessing, to assess the quality of the images received. In addition,the processor deploys an algorithm to ensure images sent/received arefor the correct description of the item (106) (i.e., object detectionand comparison of the detected object to any descriptions received). Inaddition, the system deploys an algorithm to classify the imagesaccording to the different views (such as Front, Side, and Back views),and order the images (108). The system can provide confidence of theclassification of the images, confidence that the description iscorrect, and/or that the order of the images is correct. If one or moreof these indications is low, the system can prompt manual review for thelow confidence items (112). Finally, the system can provide the resultsof the assessments and algorithms to an automated catalogue management(110).

FIG. 2 illustrates an exemplary flowchart of a disclosed process forassessing the quality of an image 202 (or images, depending on aparticular configuration). As illustrated, the image 202 is received andtwo separate processes are performed. On the left is illustratedperforming a structural similarity analysis 204, with the result being astructural similarity score

206. On the right is illustrated adding noise/distortion 208 to thereceived image, resulting in multiple images 210, 212, 214 which arederivatives of the original image 202. Using the original image 202 andthe derivative images 210,212,214, features are extracted 216. Ingeneral, the features extracted 216 are features common across all themodels 202, 210, 212, 214 (both distorted and original), however in someconfigurations the features extracted can be identified as associatedwith only the original 202 and not found in any of the distorted images210, 212,

214. The resulting extracted features 216 and the structural similarityscore 206 are then applied to a regression model 218, such as a ridgeregression model, which can be used to assess the quality of the image202 to a standard.

FIG. 3 illustrates an exemplary convolutional neural networkarchitecture for image orientation classification. In this example,images 302, 304, 306 of an object are received from distinct angles. Asillustrated, the images are of a shirt as viewed from the back 302, side304, and front 306. These images are sent to a CNN feature extractor308, such as a CNN feature extractor (e.g., a MobileNet or otherstreamlined architecture which uses depth-wise separable convolutions tobuild neural networks), or a custom CNN feature extractor. The kernelPCA (Principal Component Analysis) features 312 are extracted along withimage embeddings 310.

These outputs 312, 310 are then input to a respective classifier lossfunction 314, which can determine the image orientation based on theloss detected.

FIG. 4 illustrates a second exemplary method embodiment. In thisexample, the system receives a plurality of images of an item (402) andidentifies, via a processor configured to perform image analysis, andwithin each image in the plurality of images, the item (404). In someconfigurations, this identification process can be further augmentedusing metadata and/or a database of products or items. The systemperforms, via the processor, a structural similarity analysis of theitem, to yield a structural similarity score (406) and, for each imagein the plurality of images applying, via the processor, a plurality ofdistortions, such that for each image in the plurality of images aplurality of distorted images are generated (408). The systemidentifies, via the processor, within the plurality of distorted imagesassociated with each image in the plurality of images, at least onefeature (410), and applies, via the processor, a regression model to theplurality of images using the at least one feature and the structuralsimilarity score (412).

In some configurations, the method can further include ordering, via theprocessor, the plurality of images based on applying the regressionmodel to the plurality of images. Likewise, the method can furtherinclude training a convolution neural network using the at least onefeature, to yield a trained convolution neural network and using thetrained convolution neural network during the applying of the regressionmodel to the plurality of images.

In some configurations, the plurality of distortions can include a meanblur, a Gaussian blur, and a bilateral blur. In some configurations, theregression model is a ridge regression.

In some configurations, the structural similarity identifies at leastluminance, contrast, and structure of the item.

In some configurations, the plurality of images include a front image, aside image, and a back view of the item.

With reference to FIG. 5 , an exemplary system includes ageneral-purpose computing device 500, including a processing unit (CPUor processor) 520 and a system bus 510 that couples various systemcomponents including the system memory 530 such as read-only memory(ROM) 540 and random access memory (RAM) 550 to the processor 520. Thesystem 500 can include a cache of highspeed memory connected directlywith, in close proximity to, or integrated as part of the processor 520.The system 500 copies data from the memory 530 and/or the storage device560 to the cache for quick access by the processor 520. In this way, thecache provides a performance boost that avoids processor 520 delayswhile waiting for data. These and other modules can control or beconfigured to control the processor 520 to perform various actions.Other system memory 530 may be available for use as well. The memory 530can include multiple different types of memory with differentperformance characteristics. It can be appreciated that the disclosuremay operate on a computing device 500 with more than one processor 520or on a group or cluster of computing devices networked together toprovide greater processing capability. The processor 520 can include anygeneral purpose processor and a hardware module or software module, suchas module 1 562, module 2 564, and module 3 566 stored in storage device560, configured to control the processor 520 as well as aspecial-purpose processor where software instructions are incorporatedinto the actual processor design. The processor 520 may essentially be acompletely self-contained computing system, containing multiple cores orprocessors, a bus, memory controller, cache, etc. A multi-core processormay be symmetric or asymmetric.

The system bus 510 may be any of several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of bus architectures. A basicinput/output (BIOS) stored in ROM 540 or the like, may provide the basicroutine that helps to transfer information between elements within thecomputing device 500, such as during start-up. The computing device 500further includes storage devices 560 such as a hard disk drive, amagnetic disk drive, an optical disk drive, tape drive or the like. Thestorage device 560 can include software modules 562, 564, 566 forcontrolling the processor 520. Other hardware or software modules arecontemplated. The storage device 560 is connected to the system bus 510by a drive interface. The drives and the associated computer-readablestorage media provide nonvolatile storage of computer-readableinstructions, data structures, program modules and other data for thecomputing device 500. In one aspect, a hardware module that performs aparticular function includes the software component stored in a tangiblecomputer- readable storage medium in connection with the necessaryhardware components, such as the processor 520, bus 510, display 570,and so forth, to carry out the function. In another aspect, the systemcan use a processor and computer-readable storage medium to storeinstructions which, when executed by the processor, cause the processorto perform a method or other specific actions. The basic components andappropriate variations are contemplated depending on the type of device,such as whether the device 500 is a small, handheld computing device, adesktop computer, or a computer server.

Although the exemplary embodiment described herein employs the hard disk560, other types of computer-readable media which can store data thatare accessible by a computer, such as magnetic cassettes, flash memorycards, digital versatile disks, cartridges, random access memories(RAMs) 550, and read-only memory (ROM) 540, may also be used in theexemplary operating environment. Tangible computer-readable storagemedia, computer-readable storage devices, or computer-readable memorydevices, expressly exclude media such as transitory waves, energy,carrier signals, electromagnetic waves, and signals per se.

To enable user interaction with the computing device 500, an inputdevice 590 represents any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. An outputdevice 570 can also be one or more of a number of output mechanismsknown to those of skill in the art. In some instances, multimodalsystems enable a user to provide multiple types of input to communicatewith the computing device 500. The communications interface 580generally governs and manages the user input and system output. There isno restriction on operating on any particular hardware arrangement andtherefore the basic features here may easily be substituted for improvedhardware or firmware arrangements as they are developed.

The steps outlined herein are exemplary and can be implemented in anycombination thereof, including combinations that exclude, add, or modifycertain steps.

Use of language such as “at least one of X, Y, and Z” or “at least oneor more of X, Y, or Z” are intended to convey a single item Gust X, orjust Y, or just Z) or multiple items (i.e., {X and Y}, {Y and Z}, or {X,Y, and Z}). “At least one of” is not intended to convey a requirementthat each possible item must be present.

The various embodiments described above are provided by way ofillustration only and should not be construed to limit the scope of thedisclosure. Various modifications and changes may be made to theprinciples described herein without following the example embodimentsand applications illustrated and described herein, and without departingfrom the spirit and scope of the disclosure.

What is claimed is:
 1. A system comprising: one or more processors; anda memory resource storing a set of instructions, that when executed bythe one or more processors, causes the one or more processors to:receive image data characterizing an image of an item; determine aquality of the image based on one or more features extracted from theimage based on a plurality of derivative images and a structuralsimilarity score, wherein the plurality of derivative images aregenerated by applying at least one distortion to the image of the item;and determine an orientation of the image based on a detected classifierloss.
 2. The system of claim 1, wherein the quality of the image isdetermined by applying a regression model to the one or more featuresand the structural similarity score.
 3. The system of claim 2, whereinthe regression model is a ridge regression model.
 4. The system of claim1, wherein the one or more features are extracted by a convolutionalneural network.
 5. The system of claim 4, wherein the detectedclassifier loss is determined by a classifier loss function configuredto receive an output of the convolutional neural network.
 6. The systemof claim 5, wherein the output of the convolutional neural networkincludes kernel principal component analysis features and imageembeddings.
 7. The system of claim 1, wherein the at least onedistortion is one of a mean blur, a Gaussian blur, or a bilateral blur.8. The system of claim 1, wherein the structural similarity score isdetermined based on changes luminance, contrast, and structure of theimage.
 9. A computer-implemented method, comprising: receiving imagedata characterizing an image of an item; determining a quality of theimage based on one or more features extracted from the image based on aplurality of derivative images and a structural similarity score,wherein the plurality of derivative images are generated by applying atleast one distortion to the image of the item; and determining anorientation of the image based on a detected classifier loss.
 10. Thecomputer-implemented method of claim 9, wherein the quality of the imageis determined by applying a regression model to the one or more featuresand the structural similarity score.
 11. The computer-implemented methodof claim 10, wherein the regression model is a ridge regression model.12. The computer-implemented method of claim 9, wherein the one or morefeatures are extracted by a convolutional neural network.
 13. Thecomputer-implemented method of claim 12, wherein the detected classifierloss is determined by a classifier loss function configured to receivean output of the convolutional neural network.
 14. Thecomputer-implemented method of claim 13, wherein the output of theconvolutional neural network includes kernel principal componentanalysis features and image embeddings.
 15. The computer-implementedmethod of claim 9, wherein the at least one distortion is one of a meanblur, a Gaussian blur, or a bilateral blur.
 16. The computer-implementedmethod of claim 9, wherein the structural similarity score is determinedbased on changes luminance, contrast, and structure of the image.
 17. Anon-transitory computer-readable medium storing instructions, that whenexecuted by one or more processors, causes the one or more processorsto: receive image data characterizing an image of an item; extract oneor more features from the image, wherein the one or more features areextracted by a convolutional neural network; determine a quality of theimage based on the one or more features extracted from the image basedon a plurality of derivative images and a structural similarity score,wherein the plurality of derivative images are generated by applying atleast one distortion to the image of the item; and determine anorientation of the image based on a detected classifier loss based on anoutput of the convolutional neural network.
 18. The non-transitorycomputer-readable medium of claim 17, wherein the output of theconvolutional neural network includes kernel principal componentanalysis features and image embeddings.
 19. The non-transitorycomputer-readable medium of claim 17, wherein the quality of the imageis determined by applying a regression model to the one or more featuresand the structural similarity score.
 20. The non-transitorycomputer-readable medium of claim 17, wherein the at least onedistortion is one of a mean blur, a Gaussian blur, or a bilateral blur.